Benchmark scalar functions on production-sized inputs - #9729
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
02e9a11 to
931b4b1
Compare
Merging this PR will degrade performance by 79.29%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | add_i64_nonnull_avx512 |
6.8 µs | 76 µs | -91% |
| ❌ | WallTime | add_i64_nonnull_avx2 |
7.9 µs | 87.5 µs | -90.93% |
| ❌ | WallTime | add_i64_nullable_avx512 |
8.4 µs | 84.6 µs | -90.09% |
| ❌ | WallTime | compare_float_avx2 |
5.1 µs | 51.1 µs | -90.06% |
| ❌ | WallTime | add_i64_nullable_avx2 |
9.3 µs | 89.8 µs | -89.68% |
| ❌ | WallTime | compare_float_neon |
8 µs | 75.4 µs | -89.35% |
| ❌ | WallTime | lt_i64_nullable_avx2 |
3 µs | 25.9 µs | -88.61% |
| ❌ | WallTime | lt_i64_nullable_neon |
3.6 µs | 31.1 µs | -88.58% |
| ❌ | WallTime | compare_f32_avx2 |
4.6 µs | 39.1 µs | -88.25% |
| ❌ | WallTime | lt_i64_nullable_avx512 |
2.9 µs | 24.3 µs | -88.24% |
| ❌ | WallTime | compare_f32_neon |
5.3 µs | 41.5 µs | -87.14% |
| ❌ | WallTime | compare_u8_neon |
3.7 µs | 28.4 µs | -87.07% |
| ❌ | WallTime | compare_f32_avx512 |
3.6 µs | 27.2 µs | -86.95% |
| ❌ | WallTime | compare_int_eq_avx2 |
3.5 µs | 26.2 µs | -86.77% |
| ❌ | WallTime | compare_int_avx512 |
3.1 µs | 23 µs | -86.67% |
| ❌ | WallTime | compare_int_eq_avx512 |
3.1 µs | 23 µs | -86.59% |
| ❌ | WallTime | compare_u64_avx512 |
3.1 µs | 22.7 µs | -86.52% |
| ❌ | WallTime | compare_float_avx512 |
4.1 µs | 30.3 µs | -86.43% |
| ❌ | WallTime | compare_int_avx2 |
3.5 µs | 25.5 µs | -86.42% |
| ❌ | WallTime | compare_u64_avx2 |
3.9 µs | 28 µs | -85.99% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/production-sized-scalar-benchmarks (931b4b1) with develop (aaed723)
Footnotes
-
224 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
|
Seems like we need to discuss more about the state of our benchmarking so I'm closing this for now |
1 participant
Summary
Increases fixed-width scalar function benchmark inputs to the smaller of the default 1 MiB writer block and 100,000-row scan split. This makes the hot loop a larger part of each measurement while keeping the inputs representative of scan execution.
Changes
Adds a shared fixed-width sizing helper and applies it to the primitive binary operation and comparison benchmarks. The resulting cases remain well below 1 ms on CI and locally on my macbook m4 max.